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Using  Technology  to  Support  the  Army  Learning  Model 

Introduction 

A  key  component  of  the  Army  Learning  Model  (ALM)  is  the  application  of  technology  to 
enhance  learning.  ALM  indicates  that  technology  should  be  leveraged  to  “make  course  content 
more  operationally  relevant,  engaging,  individually  tailored,  and  accessible”  (TRADOC 
Pamphlet  525-8-2,  U.S.  Army  Training  and  Doctrine  Command,  2011,  p.12).  This  includes  using 
emerging  technologies  such  as  mobile  computing,  open  content,  electronic  books,  augmented 
reality,  gesture -based  computing,  and  visual  data  analysis  to  improve  learning.  The  ALM 
indicates  that  technology  should  be  used  in  conjunction  with  face-to-face  instruction  (i.e., 
blended  learning)  as  a  means  of  expediting  the  learning  process  (TRADOC,  2011).  Currently, 
the  Army  has  been  directed  to  “create  a  learning  environment  that  enables  mastery  of 
fundamental  competencies  through  an  appropriate  mix  of  live  and  technology-enabled  learning 
methods”  (U.S.  Training  and  Doctrine  Command,  2011,  p.15). 

In  response  to  TRADOC  Pamphlet  525-8-2  (201 1),  many  of  TRADOC's  Centers  of 
Excellence  (CoEs)  have  established  internal  groups  to  design  and  develop  technologies  in 
support  of  ALM.  The  present  research  examined  the  effects  of  the  development  and  use  of 
technology  products  (e.g.,  by  developers,  instructional  designers,  course  managers,  instructors, 
students)  within  the  CoEs  and  identified  best  practices  and  lessons  learned  for  incorporating 
these  products  into  courses  in  support  of  ALM.  This  work  also  supported  TRADOC  objectives 
regarding,  1)  identifying  factors  that  support  or  inhibit  the  development  and  use  of  effective 
technology  products,  and  2)  presenting  case  studies  representing  both  effective  and  ineffective 
design,  development,  and  implementation  of  technology  products. 

Method 

Senior  leaders  (e.g..  Branch  Chiefs,  Directors)  from  eight  Army  Centers  of  Excellence 
(CoE)  as  well  as  the  Training  Brain  Operations  Center  (TBOC)  identified  a  broad  and 
representative  sample  of  technology  products  from  their  respective  centers  to  include  in  the 
project  (N  =  44).  They  also  completed  product  specification  worksheets  for  the  identified 
products  which  captured  information  regarding  the  most  relevant  dimensions  and  characteristics 
of  each  product  (e.g.,  delivery  method  of  product;  knowledge/skill/ability  focus  of  the  product; 
course  supported  by  product;  single-user  vs.  multi-user/player;  was  product  implemented  in  past 
24  months). 

The  data  from  these  worksheets  were  analyzed  to  identify  a  smaller  subset  of  products  for 
a  case  study  analysis  that  varied  across  interactive  multimedia  instruction  (IMI)  levels  and  that 
were  representative  of  each  organization  and  of  the  Army’s  overall  use  of  technology  in  support 
of  ALM  (TRADOC  Pamphlet  350-70-12,  2013).  The  goal  was  to  identify  a  representative 
sample  that  included  both  highly  engaging  and  interactive  products  as  well  as  products  that 
offered  a  more  passive  experience  (i.e.,  limited  or  no  interactivity).  The  final  sample  that  was 
selected  for  case  study  analysis  consisted  of  21  products  from  seven  CoEs  (see  Appendix  A  for  a 
more  detailed  description  of  the  methodology). 
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Site  visits  were  then  conducted  at  each  CoE  to  identify  factors  that  facilitated  or 
inhibited  the  successful  development  and  implementation  of  each  of  the  CoE’s  sample  products, 
as  well  as  to  determine  the  product’s  effects  on  training  developers,  instructional  designers, 
course  managers,  instructors  and  students.  Interviews  and  focus  groups  were  conducted  with  a 
total  of  89  participants  from  across  seven  CoEs  (see  Table  1). 

Table  1 

Interview  / Focus  Group  Participants 


Positions 

N 

Training  Developers 

11 

Instructional  Designers 

4 

Course  Managers 

8 

Instructors 

21 

Students 

20 

Participants  in  more  than  one  role  (e.g.,  instructor  /  course  manager) 

17 

Other  Personnel  (e.g.,  forum  facilitator,  technical  director,  branch  chief) 

8 

Total 

89 

Results 

The  interview  responses  were  first  analyzed  by  IMI  level  to  provide  insights  into  the 
successes  and  challenges  of  developing  and  implementing  technology  products  in  CoE  courses 
specific  to  that  level  of  interactivity  (see  Appendix  A  for  the  methodology  and  results  by  IMI 
level).  Common  themes  also  were  determined  across  all  of  the  IMI  levels  and  offer  a  broader 
perspective  of  the  overall  challenges  of  developing  and  implementing  technology  regardless  of 
the  specific  level  of  interactivity  (see  results  in  Appendix  B). 

In  order  to  best  determine  the  potential  return  on  investment  (ROI)  for  the  CoEs  when 
deciding  to  develop  and  implement  a  technology  product  in  a  course,  facilitating  and  limiting 
effects  on  ROI  were  determined  for  part-task  (16  level  2  and  3  IMI  products)  and  whole  task  (5 
primarily  level  4  IMI  products)  trainers.  An  example  of  a  part-task  trainer  would  be  a  computer- 
based  training  or  mobile  application  instructing  students  on  how  to  perform  specific  tasks  in  a 
guided  manner.  A  level  4  IMI  product  can  be  a  part-task  trainer  if  it’s  a  highly  interactive 
product  yet  not  representing  the  full  functionality  of  the  equipment  (covering  all  tasks).  A  level 
4  whole-task  trainer  product  is  generally  termed  a  simulator  which  covers  all  tasks  related  to  the 
equipment.  An  example  of  a  whole  task  trainer  would  be  an  aircraft  simulator.  The  simulator 
resembles  the  actual  equipment  (hardware  and  software),  allowing  students  to  interact  with  it  to 
complete  all  tasks  related  to  that  equipment. 

Table  2  focuses  on  part-task  trainers  and  lists  both  facilitating  and  limiting  factors  to  a 
high  quality  product  in  support  of  ALM.  For  each  facilitating  and  limiting  factor  listed, 
corresponding  ROI  effects  are  described  in  detail.  The  facilitating  factors  include  front-end 
analysis,  subject  matter  expert  (SME)  involvement,  editable  software,  accurate  visuals  of 
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equipment,  creative  instructors,  and  a  train-the-trainer  approach.  The  limiting  factors  include 
lack  of  front-end  analysis  and  stakeholder  consensus,  limited  SME  involvement,  lack  of  POI 
integration  and  product  ownership,  difficulty  maintaining  the  products,  lack  of  formal  instructor 
training  on  the  products,  and  lack  of  objective  data  collected  on  training  effectiveness  or  the  need 
for  product  modifications.  Out  of  the  16  part-task  trainers  developed,  7  were  being  used  as  part 
of  a  course  as  stated  by  instructors. 

Table  2 

ROI  Effects  of  Facilitating  and  Limiting  Factors  for  Part-Task  Trainers 


PART-TASK  TRAINERS 

Facilitating  Factors 

ROI  Effects 

Front-end  &  On-going  Analyses 

Performing  front-end  analysis/ 
assessments  throughout  the  product 
development  cycle:  (1)  collaborative 
meetings  being  held  with  SMEs, 
Stakeholders  (leadership),  Training 
Developers,  Instructors,  and  Course 
Managers,  (2)  Training  Developers 
observing  classes  and  identifying 
strategies  based  on  these  observations, 
and  (3)  identifying  training  needs  via 
assessing  or  observing  students 

The  stakeholders  could  properly  identify  resources 
needed  for  the  project,  including  whether  to 
develop  the  product  in-house  or  use  a  contractor. 
Front-end  analysis  also  established  technical 
requirements  (i.e.  identifying  appropriate 
software). 

The  products  met  the  training  intent  since  training 
needs  were  identified  during  the  analysis.  This  led 
to  increased  use  of  the  product  in  the  classroom. 

Collaborative  meetings  kept  everyone  on  the  same 
page  so  that  the  end  product  met  the  stakeholders’ 
vision. 

SME  Support 

Availability  of  and  accessibility  to  the 
same  SME’s  throughout  the  product 
development  cycle 

SMEs  provided  accurate  and  relevant  input  at 
critical  points  in  the  products’  development.  They 
provided  feedback  during  the  development  cycle 
so  that  developers  could  make  appropriate 
modifications  in  a  timely  manner.  Accurate  and 
relevant  products  led  to  increased  use  in  the 
classroom.  Timely  SME  feedback  reduced 
lifecycle  costs  and  put  products  in  the  hands  of 
instructors  and  students  soon  to  better  support  the 
achievement  of  learning  and  course  outcomes. 

Technology  Specifications 

Using  editable  software  with  skilled  / 
qualified  in-house  personnel 

Editing  software  in-house  increased  the  CoE’s 
ability  to  maintain  the  products  when  changes 
were  required  rather  than  paying  for  contractor 
support.  Reduced  product  ‘down-time’. 
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Table  2  ( Continued ) 


Using  accurate  visuals  of  equipment 
including  the  inside  of  a  device  or  piece 
of  equipment  (i.e.  using  3-D  models) 

Increased  the  retention  of  information  and 
recognition  of  components  when  working  on  live 
equipment. 

Instructor  Use/Application 

Using  intuition  and  creativity  to 
implement  the  product  [despite  lack  of 
Program  of  Instruction  (POI) 
integration] 

Instructors  leveraged  the  products  in  a  variety  of 
ways  to  address  learning  objectives  (i.e.,  products 
used  for  course  preview  or  review,  homework, 
instruction,  practice,  remediation  or  refresher). 
Instructors  also  used  products  to  generate 
discussion  or  foster  peer  to  peer  learning  in  the 
classroom. 

Instructor  Training 

Using  a  train-the-trainer  approach  to 
disseminate  information  about  the 
product  and  its  best  use 

Increased  the  use  of  the  product  in  the  classroom 
and  facilitated  optimal  use/application  of  products 
as  opposed  to  instructors  learning  the  product’s 
capabilities  through  ‘trial  and  error’  or  not  using 
the  product. 

Limiting  Factors 

ROI  Effects 

Limited  Front-end  analysis 

Lack  of  front-end  analysis  (no  training 
analysis,  vague  requirements  or 
customer-dictated  requirements) 

Resulted  in  issues  with  manpower  /  resources  for 
product  development  and  sustainment  and 
maintenance  of  the  product  (e.g.,  an  in-house 
development  team  lacked  training  and  experience 
in  the  software  chosen  to  develop  the  product). 

Products  developed  were  not  instructionally 
sound,  relevant,  accurate,  sustainable  (editable), 
engaging,  or  did  not  meet  the  training  intent.  This 
led  to  a  decreased  use  of  the  product  in  the 
classroom. 

Stakeholder  Consensus 

Lack  of  stakeholder  consensus  for 
product  requirements 

Stakeholder  expectations  were  not  met.  For 
example,  leadership  may  have  envisioned  a  high- 
end  simulation,  however,  the  development  team 
produced  a  level  2  or  level  3  IMI  product  (which 
may  have  met  training  intent  and  was  within 
budget  but  did  not  meet  leadership’s  vision). 

SME  Support 

Table  2  ( Continued ) 


Limited  access  to  the  right  types  of 

SMEs  during  product  development  (i.e., 
those  with  relevant  knowledge  and 
experience) 

Decreased  the  fidelity  and  accuracy  of  scenarios 
and/or  simulated  equipment.  Lor  example,  an 
inexperienced  SME  did  not  accurately  describe 
the  tactics  for  a  specific  combat  mission  during 
scenario  development.  Thus,  the  product  was  seen 
as  inaccurate  or  obsolete  and  its  use  was  decreased 
in  the  classroom. 

Limited  access  to  SMEs  providing 
timely  input  and  feedback 

Product  development  was  delayed  during  the 
initial  prototype  development  as  well  as  post¬ 
prototype  development  when  SME  feedback  was 
required  for  finalization. 

POl  Integration 

Lack  of  integration  of  the  product  into 
the  course  POI 

Products  were  viewed  as  optional.  In  some  cases, 
instructors  were  not  aware  of  the  product,  leading 
to  decreased  use  of  the  product.  In  other  cases, 
products  were  used  in  different  ways  by  different 
instructors,  which  meant  that  students  did  not 
experience  the  product  in  the  same  manner  across 
the  course.  Lor  example,  if  an  instructor  was 
using  the  product  as  a  means  for  discussion  and 
peer  to  peer  learning,  those  students  were 
experienced  ALM  via  the  application  of  the 
product.  Another  instructor  may  have  not  used  the 
product  at  all  or  only  used  it  as  a  self-directed 
learning,  rendering  it  a  less  engaging  exercise. 

If  a  product  was  not  used  for  instruction,  then  no 
ROI  can  be  achieved  for  the  Army. 

Lack  of  ownership  of  the  product  (i.e., 
individuals  who  follow  through  with  its 
implementation  and  inquire  about  its  use 
or  need  for  modification  via  monitoring 
or  request  for  feedback) 

Without  ownership  of  the  product,  it  was  difficult 
to  communicate  its  existence  and  its  value  to  other 
instructors.  In  some  cases,  instructors  were  not 
aware  of  the  product,  how  to  access  the  product, 
and  /  or  how  to  use  the  product  (lack  of  training). 
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Table  2  ( Continued ) 


Without  monitoring,  it  was  difficult  to  ensure  the 
products  were  being  used  as  intended  for 
instruction,  were  reliable,  and  that  the  content  was 
accurate  and  relevant.  Identification  of  bugs  or 
feedback  on  product  accuracy/relevance  occurred 
via  ‘happenstance’  or  via  feedback  from 
students/instructors  resulting  in  student/instructor 
frustration  with  product. 

Technology  Specifications 

Using  software  that  was  not  editable;  or 
not  having  the  in-house  capability  to 
work  with  the  software 

Without  editable  software,  there  was  a  reliance  on 
the  contractor  to  support  changes,  which  led  to 
delays  and  issues  with  maintenance  contracts. 

When  no  maintenance  contract  was  in  place,  the 
products  were  not  updated,  thus  instructors  were 
not  using  the  products  or  students  were  using 
products  with  inaccurate  or  obsolete  information. 

Without  in-house  capability  to  update  the 
software,  delays  occurred  in  making  the  edits  or 
products  were  not  updated,  thus  instructors  were 
not  using  the  products  or  students  were  using 
products  with  inaccurate  or  obsolete  information. 

Instructor  Training 

Instructors  relied  on  other  instructors  to  train  them 
on  how  to  use  the  product  and  the  best  way  to 
integrate  the  product.  If  the  instructor  who  trained 
the  other  instructor  did  not  know  the  full  potential 
of  the  product,  then  this  information  gap  was 
passed  on  to  the  next  instructor. 

Lack  of  formal  instructor  training  on 
how  to  integrate  products  into  lesson 
plans 

Without  formal  training  on  the  products, 
instructors  did  not  use  the  products  consistently. 
This  led  to  students  not  receiving  the  same 
experience.  For  example,  some  instructors  used 
the  products  to  generate  discussion  and  peer-to- 
peer  interaction,  which  led  to  a  more  engaging 
experience  for  those  students  as  opposed  to 
students  who  used  the  product  in  a  self-directed 
manner. 

Evaluation 

No  objective  student  outcome  data 
collected  on  products 

Unable  to  determine  the  products’  training 
effectiveness,  support  of  ALM,  or  ROI. 
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Table  2  ( Continued ) 


No  formal  mechanism  in  place  to  collect 
or  capture  feedback  on  products  from 
instructors  and  students  concerning 
content  accuracy,  relevance,  or  system 
bugs  (not  explicitly  part  of  course 
critiques) 


Without  feedback,  it  was  difficult  for  individuals 
charged  with  maintaining  products  to  know  when 
content  issues  needed  to  be  addressed  and  made  it 
harder  for  them  to  fix  problems  quickly  when  they 
arose.  When  errors  and  bugs  persisted  in  a 
product  over  extended  periods  of  time,  instructors 
and  students  tended  to  not  to  use  them  (i.e., 
viewed  them  as  inaccurate  or  obsolete  products). 


Table  3  focuses  on  whole-task  trainers  and  also  lists  both  facilitating  and  limiting  factors 
to  a  high  quality  product  in  support  of  ALM  along  with  associated  ROI  effects.  These  products 
were  developed  by  contractors.  Facilitating  factors  include  the  use  of  products  as  a  way  to 
prepare  for  live  equipment  as  well  as  onsite  contractor  support  to  assist  with  training  and 
troubleshooting.  In  addition  to  similar  limiting  factors  as  found  with  part-task  trainers,  other 
limiting  factors  include  issues  with  maintenance  contracts  as  well  as  little  performance  feedback 
provided  from  the  product  itself  which  requires  instructor  monitoring  of  student  performance  and 
increases  instructor  workloads.  Out  of  the  5  whole  task  trainers  developed,  4  were  being  used  as 
part  of  a  course  as  stated  by  instructors. 

Table  3 

ROI  Effects  of  the  Facilitating  and  Limiting  Factors  for  Whole-Task  Trainers 


WHOLE-TASK  TRAINERS 

Facilitating  Factors 

ROI  Effects 

POI  Integration 

Students  were  provided  opportunities  to  practice 
before  being  tested  on  the  actual  equipment.  The 
products  helped  bridge  the  gap  between 
instruction  and  live  equipment. 

Use  of  products  as  the  “walk”  phase  of 
the  Army’s  crawl,  walk,  run  training 
method. 

Instructors  were  able  to  make  use  of  downtime  to 
provide  struggling  students  opportunities  for  more 
practice. 

When  there  was  only  a  limited  amount  of  live 
equipment  available  for  use,  instructors  were  able 
to  make  use  of  downtime  to  provide  practice 
opportunities  for  students  who  were  waiting  to  get 
on  the  live  equipment. 

Since  students  were  able  to  practice  on  the 
simulator  and  achieve  mastery,  this  resulted  in  less 
wear  and  tear  on  actual  equipment  (less  breakage), 
reduction  in  live  rounds  fired  or  less  fuel  used. 

Product  Updates 
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Table  3  ( Continued ) 


Contractor  support  onsite  and  available 
for  training  instructors  and 
troubleshooting  issues  that  arose  during 
class  time 

When  the  contractor  who  developed  the  product 
was  onsite,  the  training  which  instructors  received 
was  more  robust,  and  the  instructors  were  trained 
on  the  full  potential  of  the  product.  For  example, 
instructors  were  shown  how  to  pull  data  from  the 
simulators  to  measure  performance  and  provide 
feedback. 

During  class  time,  when  there  were  system 
shutdowns  or  other  technical  issues,  these 
problems  were  quickly  addressed,  leading  to  less 
student  wait  time  to  bring  systems  back  up  to  full 
functionality. 

Reduction  of  training  time  on  the  live 
equipment 

By  reducing  training  time  on  the  live  equipment, 
ROI  is  positively  affected.  Some  of  the  examples 
of  the  effects  on  ROI  included  a  reduction  of  wear 
and  tear  on  the  live  equipment  as  well  as  reduction 
in  fuel  costs  as  it  pertained  to  time  needed  in 
aircrafts  (since  students  were  already  prepared  and 
did  not  need  to  train  on  fundamental  tasks). 

Limiting  Factors 

ROI  Effects 

Contractor  Support 

No  maintenance  contract  in  place 

Products  were  updated  in  a  very  limited  manner 
depending  on  in-house  capabilities  or  they  were 
not  updated  at  all.  Obsolete  or  inaccurate  products 
led  to  negative  training  and  frustration  for  students 
who  are  familiar  with  the  equipment.  To  avoid 
negative  training  or  student  frustration,  instructors 
did  not  use  the  product  in  classroom. 

POl  Integration 

Lack  of  integration  into  the  course  POI 

Instructors  customized  and  adapted  scenarios  to 
meet  different  learning  objectives  in  order  to 
maximize  impact  on  training  effectiveness.  They 
had  to  use  their  own  experience  or  intuition  or  rely 
on  another  instructor  for  guidance. 

Instructor  Training 

Limited  formal  instruction  on  how  to 
integrate  the  simulator  into  the  lesson 

Instructors  used  the  simulators  in  different  ways  or 
did  not  use  simulators  to  their  potential.  This  led 
to  inconsistent  experiences  for  the  students. 

Student  Feedback 
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Table  3  ( Continued ) 


Products  provided  minimal  real-time 
feedback  to  students  regarding  their 
performance 

When  the  products  did  not  have  an  embedded 
after  action  review  (AAR),  the  instructors  were 
required  to  monitor  student  performance  using 
computer  monitors,  video  footage,  and  data 
generated  from  the  simulator  in  order  to  measure 
performance  and  provide  feedback.  This  led  to 
increased  instructor  workloads,  and  some 
instructors  were  not  monitoring  student 
performance  or  conducting  limited  monitoring. 
Further,  some  instructors  did  not  know  how  to  pull 
performance  data  from  the  simulators,  resulting  in 
students  who  did  not  receive  feedback  on  their 
performance. 

Evaluation 

No  objective  student  outcome  data  was 
collected  on  products 

Unable  to  determine  the  products’  training 
effectiveness,  support  of  ALM,  or  ROI. 

No  formal  mechanism  in  place  to  collect 
feedback  on  products  from  instructors 
and  students  concerning  content 
accuracy,  relevance,  or  system  bugs  (not 
explicitly  part  of  course  critiques) 

As  with  the  part-task  trainers,  without  feedback, 
this  made  it  more  difficult  for  individuals  charged 
with  maintaining  products  (in  this  case,  the 
contractors)  to  know  when  accuracy  issues  or  bugs 
needed  to  be  addressed.  Thus,  making  it  harder 
for  them  to  fix  problems  quickly  when  they  arose. 
When  errors  and  bugs  persisted  in  a  product  over 
an  extended  periods  of  time,  instructors  and 
students  tended  to  not  to  use  them  (i.e.,  viewed 
them  as  inaccurate  or  obsolete  products). 

Success  and  challenge  stories  are  reported  below  that  provide  exemplars  of  the 
facilitating  and  limiting  factors  presented  in  Tables  2  and  3.  The  stories  are  supported  by 
statements  made  during  interviews  and  focus  groups. 

Success  Stories 

Part-task  trainer.  After  receiving  notification  from  the  Commanding  General  that 
Soldiers  needed  to  learn  about  making  decisions  while  thinking  about  the  2nd  and  3rd  order 
effects,  an  in-house  development  team  held  a  structured  meeting  which  included  instructional 
designers  from  the  multimedia  branch  as  well  as  training  developers,  and  SMEs.  Although  the 
development  of  the  product  was  not  based  on  a  formal  analysis  (i.e.  a  training  gap  analysis),  the 
design  was  supported  by  an  informal  analysis  during  which  SMEs  supplied  scenario  ideas,  and 
the  multimedia  team  discussed  technologies  that  could  support  the  execution  of  these  scenarios 
in  the  most  effective  manner.  It  was  indicated  that  the  team  had  benefited  greatly  from  that 
initial  meeting. 

Based  on  those  discussions,  the  multimedia  team  selected  VBS2  (Virtual  Battlespace  2)  to  record 
the  scenarios  that  would  play  out  based  on  the  student’s  decision.  Using  this 
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technology  instead  of  live  video  facilitated  editing  and  allowed  the  scenarios  to  contain  more 
graphic  scenes  which  provided  the  most  impact  on  the  student.  The  in-house  team  was 
successful  in  coming  up  with  innovative  and  creative  ideas  to  support  the  needs  mentioned  in  the 
meeting. 

The  branch  had  a  process  in  place  for  requesting  SMEs,  so  they  were  able  to  obtain 
dedicated  SMEs.  The  SMEs  continued  to  work  with  the  multimedia  team  throughout  the 
development  of  the  product  and  were  capable  of  providing  specific  input  required  for  the 
development  of  scenarios. 

Part-task  trainer.  This  product  was  installed  on  Blackboard,  allowing  for  the  usage 
tracking  of  the  training  product.  As  with  other  products  that  were  examined,  this  product  was 
designed  to  support  the  “walk”  phase  of  the  “crawl,  walk,  run”  cycle.  This  allows  instructors  to 
observe  the  transfer  of  skills  when  students  are  working  on  live  equipment.  It  was  indicated  that 
there  was  a  good  transfer  from  the  training  product  to  the  actual  use  of  the  equipment  and  that 
students  were  more  confident  conducting  training  on  the  live  equipment.  Although  knowledge 
and  skills  transfer  data  was  not  formally  collected  via  assessment  scores,  the  product  developed 
to  support  the  “walk”  phase  allowed  for  performance  evaluation  on  live  equipment,  which 
reflects  a  level  three  Kirkpatrick  evaluation  (Kirkpatrick,  1994,  2006). 

Whole-task  trainer.  This  product  was  implemented  after  instructors  received  training 
from  the  product  developers  (contractors  in  this  case).  The  product  used  a  station  for  instructors 
to  monitor  the  students’  use  of  the  product  during  classroom  time  in  order  to  provide  feedback 
and  support.  The  training  the  instructors  received  included  how  to  set  up  and  use  the  monitoring 
stations  to  see  how  the  students  were  doing  and  help  when  needed.  The  instructors  could  also 
inject  faults  into  a  team’s  scenario  to  assess  troubleshooting  skills,  and  the  instructors  were 
trained  on  how  to  do  this  by  the  contractor.  The  multi-user  technology  product  supported  peer- 
to-peer  interaction  as  this  was  noted  by  instructors  as  they  moved  between  monitoring  stations 
and  observed  students  helping  other  students  during  the  activities.  The  contractors  who 
developed  the  product  are  on  site  and  available  to  troubleshoot  any  issues  that  arise. 

Challenge  Stories 

Web-based  training  application.  This  particular  IMI  product  was  developed  due  to  a 
need  to  revise  components  of  the  course.  The  Course  Manager  and  instructors  decided  to  use  an 
IMI  product  to  instruct  a  particular  portion  of  the  course.  This  was  the  extent  of  the  analysis  to 
create  a  technology-based  solution.  They  outsourced  this  to  a  contractor  and  did  not  discuss 
with  the  contractor  what  level  of  interactivity  was  envisioned.  It  was  indicated  that  the 
conversation  with  the  contractor  would  have  been  different  had  ALM  been  implemented  at  that 
time.  Due  to  contract  delays,  the  instructors  created  PowerPoint  slides  as  an  interim  solution. 
When  a  new  vendor  was  contracted,  their  team  did  a  straight  conversion  of  those  PowerPoint 
slides.  Since  this  was  a  basic  IMI,  instructors  did  not  require  training  on  how  to  incorporate  this 
into  the  classroom;  however,  the  product’s  usage  by  instructors  diminished  over  time.  Currently, 
the  IMI  is  not  being  used  as  the  primary  method  of  instruction;  it  is  now  a  secondary  source  with 
lecture  being  used  to  teach  the  concepts.  It  was  reported  that  the  students  who  did  use  the  IMI 
seemed  to  not  have  understood  the  IMI  content. 
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Mobile  application  (job  aid).  This  particular  product  was  designed  as  a  support  tool  for 
students.  Although  the  product  was  interactive  and  was  designed  to  be  “just  in  time”  training 
support  through  use  of  mobile  devices,  this  product  was  not  integrated  into  the  course  POI  and 
viewed  as  optional.  The  product  was  designed  by  skilled  contractors  who  had  gaming  and 
mobile  technology  expertise  which  led  to  engaging  products  in  support  of  ALM.  However, 
students  were  not  aware  of  the  product  unless  they  were  told  about  it  by  other  students,  by  an 
instructor  who  saw  value  in  it,  or  if  they  saw  it  could  be  downloaded  from  Milsuite.  Furthermore, 
its  usage  was  not  tracked  by  instructors;  therefore,  students  who  were  aware  of  the  product  knew 
it  was  optional. 

Mobile  application  (job  aide).  For  this  product,  usage  or  student  performance  data  were 
not  able  to  be  collected  or  tracked  because  the  products  did  not  reside  on  a  learning  management 
system  (LMS).  Further,  there  was  no  formal  follow  up  as  to  whether  a  student  performs  better 
after  using  these  support  tools.  The  feedback  was  only  provided  via  an  email  box  which  was  set 
up  to  gather  reaction-based  comments  (with  some  content-based  comments);  these  were  then 
reviewed  by  the  development  team.  No  means  existed  to  demonstrate  training  effectiveness  with 
hard  data,  thus,  the  ability  to  determine  the  effects  of  using  the  technology  for  Army  training  was 
limited  (i.e.,  limited  ability  to  determine  ROI  effects). 

Discussion  and  Conclusions 

A  cross-walk  of  the  factors  for  the  part-task  and  whole-task  trainers  indicated  that  none  of 
the  facilitating  factors  overlapped.  This  may  be  due  in  part  to  the  idea  that  Army  personnel  (both 
military  and  civilians)  were  actively  involved  in  the  planning  for  and  development  of  the  part- 
task  trainers.  Thus,  many  of  the  facilitating  factors  appeared  to  be  related  to  the  activities 
involved  in  planning  and  developing  the  products  (e.g.,  performing  analyses,  availability  and 
accessibility  to  SMEs,  using  editable  software  with  skilled/qualified  in-house  personnel).  On  the 
other  hand,  for  the  whole-task  trainers,  the  front-end  analysis  is  typically  conducted  as  part  of  a 
larger  Army  material  acquisition  process.  In  this  case,  comprehensive  DOTMLPF  (Doctrine, 
Organization,  Training,  Material,  Leadership  &  Education,  Personnel,  and  Facilities)  studies  are 
conducted  which  feed  into  the  JCIDS  (Joint  Capabilities  Integration  Development  System) 
process.  The  individuals  who  participated  in  this  research  may  not  have  taken  part  in  that 
process  or  assumed  their  positions  long  after  these  processes  occurred.  Consequently,  the 
facilitating  factors  for  the  whole-task  trainers  reflected  the  use  of  the  products  while  executing 
the  course  (e.g.,  Use  of  products  as  the  “walk”  phase  of  the  Army’s  crawl,  walk,  run  training 
method,  contractor  support  on  site)  and  the  positive  effects  on  ROI  by  reducing  the  amount  of 
training  time  needed  on  live  equipment. 

The  limiting  factors  that  potentially  affect  ROI  were  essentially  the  opposite  of  the 
facilitating  factors.  That  is,  if  no  front-end  analysis  was  conducted  for  the  part-task  trainers,  then 
this  potentially  has  negative  effects  on  the  ROI  in  that  the  products  developed  were  more  likely 
to  be  inaccurate  or  not  instructionally  sound  and  thus  not  used  in  the  classroom.  For  the  whole- 
task  trainers,  one  limiting  factor  is  the  lack  of  a  maintenance  contract  which  is  the  opposite  of 
having  onsite  contractor  support.  Thus,  there  were  many  more  limiting  factors  for  the  part-task 
trainers  due  to  the  many  different  issues  that  arose  during  the  planning  and  the  development  of 
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the  products  often  using  in-house  capabilities  and  resources.  For  the  whole-task  trainers,  the 
planning  was  likely  accomplished  with  very  large  teams  of  personnel  and  the  development 
accomplished  by  contractors  offsite  and  delivered  to  the  Government. 

One  limiting  factor  that  was  identified  for  both  product  categories  was  a  lack  of  instructor 
training.  A  total  focus  on  the  technology  development  will  not  necessarily  lead  to  successful 
implementation  or  high-level  of  use  in  the  classroom.  A  systematic  process  for  training  new 
instructors  on  how  to  use  the  full  capabilities  of  the  products,  how  to  best  use  the  products  to 
support  the  course  outcomes,  and  how  to  best  address  challenges  that  occur  when  students  are 
using  the  products  is  critical  to  maximize  the  effectiveness  and  ROI  of  the  products. 

When  the  facilitating  factors  were  compromised  and/or  the  limiting  factors  were  present, 
challenges  arose  for  these  products — ultimately  leading  to  less  effective  use  of  the  product, 
decreased  use  of  the  product,  or  even  no  use  of  the  product  in  the  classroom.  The  findings 
indicated  that  44%  (n  =  7/16)  part-task  versus  80%  in  =  4/5)  whole  task  trainers  were  being  used 
in  the  classroom  or  to  support  some  other  learning  event.  Nearly  all  of  the  whole-task  trainers 
were  being  used  in  the  classroom  despite  the  lack  of  a  maintenance  contract  in  some  cases 
(indicating  a  lack  of  updates  to  keep  the  product  relevant  and  current).  Thus,  although  the  front- 
end  costs  of  developing  whole-task  trainers  can  far  exceed  those  of  the  part-task  trainers,  these 
products  have  a  much  better  chance  of  being  sustained  in  part  due  to  the  extensive  front-end 
analyses  that  are  conducted  to  obtain  requirements,  SME  involvement,  systematic  reviews  to 
ensure  requirements  are  met,  and  often  POI  integration. 

Further,  the  factors  for  the  part-task  trainers  leading  to  positive  outcomes  are  closely 
aligned  with  a  successful  employment  of  the  ADDIE  (Analysis,  Design,  Development, 
Implementation,  and  Evaluation)  model  for  training-based  interventions.  Those  products  which 
were  developed  based  on  a  front-end  analysis  with  SME  input  and  supported  by  effective 
instructor  training  and  communication  were  used  more  in  the  classroom  and  used  in  ways  which 
support  ALM.  As  evidenced  in  the  success  stories  above,  when  the  Army  employs  the  ADDIE 
model  to  the  fullest  extent  possible  during  instructional  materials  development  or  revisions,  the 
outcome  leads  to  a  solution  which  supports  the  intent,  in  this  case  supports  ALM.  If  the  intent  is 
to  produce  instructional  material  or  a  product  that  supports  Soldier-centered  training,  peer-to- 
peer  collaboration,  or  critical  thinking  skills,  among  other  initiatives,  adhering  to  the  ADDIE 
model  will  generate  the  best  outcome  for  this  intent.  Those  products  which  were  developed 
without  or  with  little  front-end  analysis  and  limited  SME  input  were  at  a  higher  risk  for  being 
inaccurate,  obsolete,  and  unable  to  meet  the  training  intent,  leading  to  decreased  or  no  use  in  the 
classroom. 

For  both  categories  of  products,  a  significant  finding  was  that  little  to  no  objective 
student  performance  data  was  collected  to  verify  that  the  products  met  the  training  intent, 
improved  Soldiers’  performance,  and  supported  all  or  some  of  ALM  tenets.  Due  to  this  lack  of 
objective  data,  the  CoEs  and  the  Army  lack  a  vital  means  of  gauging  whether  these  technology- 
based  training  tools  are  supporting  the  successful  implementation  of  ALM.  In  terms  of  ROI, 
when  the  evaluation  phase  of  the  model  is  not  executed  at  all  (or  even  in  part),  it  is  challenging  to 
determine  the  extent  to  which  the  instructional  material  or  product  was  a  sound  investment.  This 
creates  a  challenge  for  sustainment  and  considerations  of  manpower  and  resources  for  the  future. 
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A  plan  that  specifies  how  the  evaluation  will  be  conducted  and  what  resources  are  needed  to 
conduct  an  evaluation  of  the  product  is  needed  to  be  agreed  upon  by  stakeholders  at  the 
beginning  of  the  technology  product  planning  and  development  process.  “After  the  fact” 
evaluations  typically  do  not  give  the  decision-makers  the  information  they  need  or  desire  (e.g., 
no  baseline  information,  subjective  data  from  instructors  regarding  student  learning  with  and 
without  the  technology). 
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Appendix  A: 


Research  Method 

The  purpose  of  this  project  was  to  identify  and  examine  a  sample  of  technology-based 
training  products  developed  and/or  implemented  by  seven  Centers  of  Excellence  (CoEs) 1  and  the 
Training  Braining  Operations  Center  (TBOC)  in  an  effort  to  identify  lessons  learned  and  best 
practices  for  incorporating  these  products  into  courses  to  support  the  Army’s  new  Learning 
Model  (ALM).  This  was  accomplished  in  two  stages. 

Stage  I:  Sampling  of  Training  Products 

CoEs  were  tasked  with  identifying  a  representative  sample  of  technology-based  training 
products  that  were  either  developed  for  and/or  implemented  in  a  course  at  their  respective 
centers.  For  each  product  identified  they  completed  a  product  specification  worksheet  that 
captured  the  most  relevant  dimensions  and  characteristics  of  each  product  (see  Figure  1).  Once 
all  worksheets  were  returned,  follow-up  conference  calls  were  scheduled  with  each  CoE  to 
address  information  gaps  in  the  worksheets  and  to  request  copies  of  each  product,  as  well  as  any 
supporting  materials  that  might  have  been  available  (e.g.,  user  guides,  training  manuals).  During 
these  efforts,  44  technology-based  solutions  were  selected  for  further  study,  with  each 
organization  contributing  approximately  three  to  six  products. 

The  information  from  the  products  was  analyzed  to  identify  how  each  of  the  technology- 
based  training  products  varied  across  the  dimensions/characteristics  captured  within  the  product 
specifications.  The  goal  was  to  identify  a  diverse  set  of  products  that  best  represented  each 
organization  and  that  varied  across  interactive  multimedia  instruction  (IMI)  level.  This  included 
both  highly  engaging  and  interactive  products  as  well  as  those  that  offered  more  of  a  passive 
experience  (i.e.,  limited  or  no  interactivity).  Based  on  this  analysis,  a  representative  sample  of 
technology-based  training  products  were  selected  from  each  organization  for  inclusion.  A  total 
of  21  products  were  selected  for  the  final  sample,  with  each  organization  contributing 
approximately  three  technology-based  training  solutions. 


1  The  Mission  Command  Center  of  Excellence  (MSCoE)  at  Ft  Leavenworth,  Kansas  was  excluded  from  this  effort.  TBOC  also  was  excluded 
from  the  data  collection  due  to  its  unique  role  as  a  resource  for  each  of  the  CoEs. 
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ATTG 

SUBJECT:  TRADOC  TASKORD  IN140362  Army  Research  Institute  Study  on  Using 
Technology  to  Support  the  Army  Learning  Model 


Product  Specification  Worksheet 

PRODUCT  NAME 

SHORT  SUMMARY  OF  PRODUCT  (i.e..  goal  of  the  moduct  eaD  product  is  intended  to  fill): 

Evaluative  Dimensions 

Response  (Responses  ere  not  limited  to  the  space) 

1)  Please  specify  the  delivery  method  product 
type  For  example,  it  may  be  a  game-based 
training,  simulation1  virtual  scenario,  traditional 
e-leaming.  virtual  classroom,  etc  ).  The 
training  may  also  be  desktop  or  mobile. 

2)  Please  describe  the  skill  ability  focus  of  this 
product  and  or  learning  gaps  addressed  by  the 
product.  Does  the  product  focus  on  the 
acquisition  of  declarative  knowledge  (terms, 
facts,  rules  concepts),  procedural  or 
demonstrative  knowledge  (how  to  perform  a 
procedure),  decision-making  or  critical  thinking, 
or  psychomotor  skills  (how  to  perform  physical 
tasks  such  as  manipulate  a  piece  of  equipment). 

3)  In  winch  course(s)  is  the  product  currently 
being  used? 

4)  Is  the  product  single-user  or  multi-user  player 
(i.e..  virtual  environment  in  which  multiple 
players  can  interact  with  one  another)? 

5)  Does  the  product  include  an  instructor 
authoring  tool?  If  so,  is  the  interface  for  this 
authoring  tool  accessible  to  and  easy  to  use  for 
instructors? 

6)  Does  the  product  include  a  complete 
instructional  cycle  (e  g.,  pre-assessment, 
didactic  portion,  demonstration,  practical 
exercise  (scenario  simulation),  post-assessment 
and  After  Action  Review)?  If  not.  which 
component  is  missing  from  this  cycle'’ 

7)  Was  the  product  implemented  m  the  past  24 
months? 

8)  Was  this  product  developed  in-house?  (If  not 
who  developed  the  product?  Who  is  maintaining 
it?) 

9)  Please  describe  design  and  development 
constraints  faced  such  as  time,  funding,  etc. 

10)  Please  describe  challenges  faced  m 
implementation  of  the  product  (may  be 
technology-related  or  any  challenges  in  general). 

UNCLASSIFIED 

Figure  Al.  Product  Specification  Worksheet. 

Stage  II:  Data  Collection 

The  research  team  then  conducted  data  collections  at  each  CoE  to  identify  best  practices, 
lessons  learned,  and  challenges  associated  with  the  development  and  implementation  of  each 
technology-based  training  product,  as  well  as  the  effects  of  these  products  on  the  curriculum, 
training  developers,  instructional  designers,  course  managers,  instructors,  and  students.  The 
following  research  questions  were  examined. 
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1.  How  well  equipped  are  the  technology-based  training  development  organizations 
(TTDOs)  for  developing  training  solutions  that  support  ALM? 

2.  What  was  the  impact  of  products  on  curriculum? 

3.  What  was  the  impact  of  products  on  instructors? 

4.  What  was  the  impact  of  products  on  students? 

5.  What  was  the  impact  of  the  product  on  course  managers/leaders? 

6.  What  was  the  impact  of  the  product  on  training  developers/instructional  designers? 

Participants.  A  sample  of  89  participants  from  the  seven  Army  CoEs  were  interviewed 
or  participated  in  focus  groups.  This  sample  included  training  developers,  instructional 
designers,  course  managers,  instructors,  students  and  other  personnel  (e.g.,  forum  facilitator, 
technical  director,  branch  chief)  experienced  in  or  involved  with  the  development  and/or 
implementation  of  technology-based  training  solutions  at  their  centers.2  Table  1  displays  the 
number  of  participants  by  category. 

Procedure.  Interviews  and  focus  groups  were  conducted  at  each  CoE  and  lasted 
approximately  60  minutes.  All  participants  received  informed  consent  forms  to  review  prior  to 
being  introduced  to  the  project  and  their  rights  as  participants  were  explained  (e.g.,  voluntary 
participation).  Participants  were  then  asked  a  series  of  questions  based  on  their  experience  with 
a  particular  technology  product  using  a  semi-structured  interview  protocol.  The  protocol  was 
divided  into  the  following  four  primary  sections  and  consisted  of  38  questions. 

1.  Description  of  the  technology-based  training  product  and  the  ALM  concepts  it  supported 
(3  questions).  For  example,  “Please  describe  the  instructional  delivery  method 
(technology-based  or  otherwise)  explain  its  use  in  the  course”  and  “Which  of  the 
following  ALM  concepts  are  supported  by  this  instructional  delivery  method?” 

2.  Development  of  the  product  to  include  the  process  used  to  identify  the  need  for  the 
product  (5  questions).  For  example,  “What  course  learning  objectives  are  addressed  by 
this  instructional  delivery  method?”  and  “To  what  degree  were  students,  instructors, 
technical  experts,  job  incumbents,  and  other  SMEs  involved  in  the  development  of  this 
instructional  delivery  method?” 

3.  Implementation  of  the  product  and  its  impact  on  those  involved  in  the  process  (11 
questions).  For  example,  “How  prepared  were  you  to  implement  the  new  instructional 
delivery  method  once  it  was  inserted  into  the  course?”  and  “How  does  this  instructional 
delivery  method  impact  students?” 

4.  Challenges,  lessons  learned,  and  best  practices  associated  with  the  product  (5  questions). 
For  example,  “What  were  some  of  the  significant  challenges  faced  in  developing  / 
implementing  /  maintaining  this  instructional  delivery  method?”  and  “From  your 
experience,  what  are  some  best  practices  for  the  future  when  developing  /  implementing  / 
maintaining  an  instructional  delivery  method  in  support  of  the  ALM?” 

In  addition  to  the  four  main  sections,  there  was  a  section  set  aside  specifically  for 
students  which  consisted  of  many  of  the  same  questions  found  in  other  parts  of  the  protocol  (14 
questions).  For  example,  “Now  that  you’ve  had  a  chance  to  use  the  instructional  delivery 


-  Please  note  that  there  were  17  participants  involved  who  held  more  than  one  role  (e.g.,  instructor/course  manager,  training 
developer/instructional  designer) 
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method,  has  it  met  your  expectations?”  and  “Does  using  this  instructional  delivery  method  help 
you  to  feel  more  engaged  in  the  training?”  Figure  2  provides  a  copy  of  the  protocol  used  during 
the  data  collection. 

Analysis.  A  content  analysis  was  conducted  using  the  Analysis,  Design,  Development, 
Implement,  Evaluate  (ADDIE)  model  as  a  framework  to  help  structure  theme  identification.  A 
five  step  process  was  used. 

Step  1.  Transcripts  were  sorted  by  product  IMI  level.  IMI  level  in  this  context  refers  to 
the  level  of  interactivity  between  the  product  and  the  learner;  with  level  1  being  the  lowest  level 
of  interactivity  and  4  being  the  highest.  A  level  1  product  is  often  referred  to  as  a  “page  turner” 
due  to  the  passive  nature  of  the  learning  and  Level  4,  on  the  other  hand,  consists  of  products  that 
provide  a  much  more  immersive  experience  for  the  learner.  Table  A-2  displays  the  number  of 
technology  products  included  in  the  analysis  for  each  IMI  level. 

Table  A1 

Products  by  IMI  Level 


IMI  Level 

N 

Level  2 

7 

Level  3 

9 

Level  4 

5 

Total 

21 

Step  2.  A  large  sample  of  interviews  and  focus  group  transcripts  was  reviewed  and 
content  analyzed  to  identify  thematic  categories  or  codes  within  each  IMI  level  according  to  each 
phase  of  the  ADDIE  model. 


Analysis,  Design  & 
Development 


SME  Related 
Front-End  Analysis 
Leadership  Support 
Technology 
Specification 


r - 1 

Implementation 

i 

r 

•  POI  Integration 

•  Instructor  Training 

•  Instructor  Use 

•  Product  Use 

\ 

^ _ 

J 

Evaluation  & 
Sustainment 


Evaluation/Learning 

Outcomes 

Product 

Update/Continuous 

Improvement 


Common  Themes  Across  Phases:  Analysis,  Design,  &  Development,  Implementation,  and  Evaluation  &  Sustainment 


H. 


•Manpower/Resources 


Figure  A-2.  Common  theme  categories  identified  for  each  phase  of  ADDIE  model. 
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Step  3.  The  thematic  categories/codes  described  above  were  then  used  to  sort  the  content 
in  each  transcript.  Coding  was  conducted  at  the  product  level  to  control  for  methodological 
issues  that  could  potentially  arise  from  coding  at  the  participant  or  session  level. 

Step  4.  Coded  themes  were  then  reviewed  for  each  product  by  IMI  level  in  order  to 
identify  those  that  were  common.  A  theme  was  considered  common  when  it  was  found  in  at 
least  50%  of  products  for  a  given  IMI  level.  For  example,  if  a  theme  was  identified  in  5  of  9 
level  3  products  (i.e.,  56%),  it  was  counted  as  a  common  for  that  level.  The  results  from  this 
effort  are  found  in  Appendix  B. 

Step  5.  Common  themes  for  each  IMI  level  were  then  reviewed  to  identify  common 
themes  across  IMI  levels  (see  Appendix  B). 
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Appendix  B: 


Cross-Cutting  Themes  and  Themes  by  IMI  Level 

As  mentioned  in  the  executive  summary  report,  we  analyzed  products  grouped  within 
their  respective  IMI  levels  before  extracting  common  themes  across  all  products.  In  this 
appendix  we  provide  a  detailed  list  of  the  cross-cutting  themes  as  well  as  a  list  of  themes  sorted 
by  IMI  level.  These  themes  are  presented  in  accordance  with  the  phase  of  the  ADDIE  model. 
These  themes  were  extracted  using  our  methodology  as  described  in  Appendix  A.  These  themes 
also  were  used  to  develop  the  facilitating  and  limiting  factors  potentially  affecting  ROI. 

Cross-Cutting  Themes 


Analysis,  Design,  Development 

Analysis  prior  to  product  design  /  development.  The  results  indicated  that  training 
developers  did  not  perform  or  performed  very  little  front-end  analysis  before  the  products  were 
designed.  Front-end  analysis  can  include  various  steps  depending  on  the  desired  outcome  yet 
most  commonly  includes  a  needs  assessment  which  identifies  the  learning  gaps  to  be  addressed 
by  the  intervention.  Different  organizational  resources  are  required  during  the  analysis  phase 
(i.e.  stakeholders,  developers)  to  determine  what  these  gaps  are  and  the  best  manner  to  address 
them.  For  course  revisions,  often  the  gaps  were  already  identified  during  the  development  of  the 
original  course;  therefore,  the  analysis  may  focus  more  on  revising  the  instructional  approach 
(i.e.  delivery  method). 

When  there  is  no  or  little  up-front  analysis,  there  is  a  risk  that  training  interventions  are 
not  systematically  aligned  with  identified  learning  objectives.  Furthermore,  if  it  is  the 
institution’s  intent  to  roll  out  new  instructional  approaches  (such  as  supporting  the  AFM),  front- 
end  analysis  is  essential  to  ensure  the  design  supports  the  new  or  different  instructional  approach. 

Front-end  analysis  should  include  a  team  meeting  to  discuss  the  learning  gaps  and 
potential  interventions.  This  allows  a  consensus  to  be  reached  on  the  best  options  and  to  plan  for 
resources,  manpower,  implementation,  and  sustainment.  Without  this  type  of  meeting,  an 
intervention  may  be  developed  that  does  not  meet  the  intent,  withstanding  issues  of  resources, 
manpower,  implementation,  and  sustainment. 

The  results  demonstrated  a  few  best  practices  that  certain  CoEs  had  implemented.  In  one 
case,  there  was  an  up-front  collaborative  meeting  with  stakeholders,  senior  leaders,  and 
developers  to  evaluate  the  training  need,  explore  options,  and  discuss  product  requirements.  In 
another  case,  training  developers  observed  classroom  instruction  prior  to  the  product 
development  in  order  to  determine  the  best  instructional  approach  for  the  POI  revision. 

Access  to  and  availability  of  SMEs.  Development  of  instructional  materials  depends 
largely  on  information  provided  by  an  individual  (or  group  of  individuals)  who  has  expert 
knowledge  of  the  subject  matter  or  proven  expertise  in  the  task.  During  the  development  phase, 
access  to  these  individuals  is  crucial  since  lack  of  access  can  stall  the  development  cycle  and  can 
threaten  the  fidelity,  accuracy,  and  relevancy  of  the  training  intervention.  The  results  suggested 
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that  access  to  SMEs  or  to  the  same  SMEs  throughout  the  development  cycle  was  a  recurring 
challenge.  SMEs  who  provided  valuable  input  and  demonstrated  dedication  to  the  project  were 
often  sought  after  by  other  organizations,  thus  inaccessible  from  the  onset  or  becoming 
inaccessible  in  the  middle  of  the  development. 

On  the  other  hand,  the  results  indicated  that  when  there  was  a  process  in  place  to  access 
and  maintain  the  same  SMEs  throughout  design  and  development,  the  outcome  met  the  desired 
intent. 


Manpower  and  resource  constraints.  As  previously  mentioned,  issues  of  manpower 
and  resources  can  be  the  result  of  a  lack  of  analysis  pertaining  to  the  intervention  requirements. 

If  a  team  does  not  collaboratively  discuss  intended  outcome,  budget,  and  technology 
requirements,  the  development  team  can  face  issues  of  manpower  and  budget  to  complete  the 
product.  The  results  suggested  fundamental  issues  of  manpower  and  resources  for  both  in-house 
and  contractor  development.  For  in-house  development,  there  appeared  to  be  a  lack  of 
manpower  to  support  technology-related  projects  as  well  as  issues  related  to  the  qualifications 
and  experience  of  those  working  on  technology  development.  When  a  contract  was  used  to 
develop  the  product  and  a  requirement  for  the  maintenance  of  the  product  was  not  specified  then 
challenges  were  reported  in  obtaining  the  necessary  updates  to  keep  the  product  current  and 
relevant. 

The  results  indicated  a  best  practice  of  using  development  software  that  is  editable  in- 
house  (i.e.  Unity  which  is  open  source  and  html-based  code).  This  requires  in-house  training 
developers  to  be  skilled  in  working  with  the  software.  Contractors  who  used  Unity  to  develop 
the  training  product  allowed  for  a  maintenance  strategy  in-house  rather  than  relying  on  the 
contractors  to  make  any  necessary  updates. 

Implementation 

POI  Integration.  As  mentioned  above,  the  lack  of  front-end  analysis  leads  to 
interventions  that  are  not  systematically  aligned  with  learning  objectives.  This  result  was  evident 
in  the  findings  that  many  products  were  not  incorporated  into  course  POIs  and  were  often 
considered  optional  instructional  tools  or  ones  in  which  the  instructors  used  in  a  variety  of  ways 
to  achieve  learning  objectives.  If  product  use  is  not  enforced,  this  leads  to  an  inconsistency  in 
the  degree  to  which  students  experience  the  ALM  instructional  approach.  If  use  of  the  products 
is  not  standardized,  and  instructors  use  them  in  different  ways,  then  there  is  a  risk  that  students 
are  not  experiencing  ALM  in  the  same  manner  or  not  experiencing  all  of  the  ALM  tenets.  For 
example,  one  instructor  may  use  the  product  as  a  way  to  facilitate  peer  to  peer  interaction  and 
instruction,  whereas  another  instructor  may  use  it  only  for  self-directed  learning.  The  students  in 
these  two  classes  are  not  having  the  same  experience  with  ALM. 

On  the  other  hand,  it  was  interesting  that  some  instructors  were  proactively  using 
products  that  were  optional  even  if  they  were  using  them  in  different  ways.  This  reveals  that 
instructors  found  the  products  useful  in  supporting  learning  objectives  even  if  not  tied  to  the  POI. 
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Instructor  training.  The  amount  of  training  required  for  the  instructors  varied  by 
product,  since  some  products  required  more  training  on  the  actual  use  of  the  product  (i.e.  IMI 
level  4  products  /simulators)  while  others  required  training  on  how  to  incorporate  the  product 
into  the  course.  IMI  level  2  products  generally  did  not  require  instructor  training  since  these 
products  were  mostly  self-explanatory  (web-based  eLearning).  However,  it  was  evident  in  the 
data  that,  even  in  cases  when  training  was  required,  it  mostly  did  not  occur  in  a  formal  or 
structured  manner  or  did  not  occur  at  all.  When  instructors  are  not  trained  on  a  product  they  may 
spend  class  time  attempting  to  work  with  the  product  and  thus  lose  valuable  instruction  time. 
They  may  also  not  use  the  product  to  its  fullest  potential,  thus  not  providing  the  maximum 
benefit  of  the  product’s  capabilities  to  the  students.  Finally,  if  the  instructors  do  not  know  how  to 
use  the  product  they  will  likely  avoid  using  it  at  all,  especially  if  use  of  the  product  is  not 
incorporated  into  the  POI. 

When  instructors  are  trained  informally  on  a  product  (i.e.  instructors  may  give  other 
instructors  a  run-through),  they  are  at  least  “brought  up  to  speed”  on  the  product’s  capabilities; 
however,  this  can  lead  to  an  inconsistency  in  how  instructors  are  trained.  An  instructor  is  only  as 
knowledgeable  about  the  product  as  the  instructor  who  showed  him  or  her.  If  that  instructor  is 
does  know  the  product’s  full  functionality,  the  other  instructor  also  will  not  be  aware  of  the 
product’s  full  capabilities.  Likewise,  another  instructor  may  end  up  using  it  to  its  fullest 
potential  since  he  or  she  was  taught  by  a  different  instructor  who  was  more  familiar  with  the 
product.  This  varied  use  of  the  product  affects  the  students,  such  that  there  is  no  consistency  in 
how  the  students  experience  the  product  or  ALM. 

The  results  indicated  that  the  products  developed  by  contractors  sometimes  included 
training  support,  and  instructors  were  more  formally  trained  on  the  product.  This  was  the  case 
mostly  with  IMI  level  4  products  that  included  instructor  monitoring  stations  and  the  need  for 
instructors  to  bring  up  systems  and  troubleshoot  issues. 

Product  use.  If  products  were  not  systematically  incorporated  into  the  course  POIs  and 
instructors  were  not  trained  on  how  to  use  the  product,  then  the  results  indicated  that  the 
instructors  used  the  products  in  different  ways  to  support  the  instruction  of  specific  learning 
objectives.  In  a  sense,  benefits  to  the  students  relied  on  the  creativity  of  the  instructor.  For 
example,  some  instructors  used  the  products  to  preview  or  review  the  material  in  the  lesson,  such 
as  during  homework,  as  actual  instruction  (in  place  of  using  lecture  and  slides),  or  during 
downtime  in  class.  For  instance,  if  students  were  waiting  to  use  live  equipment,  they  were  able 
to  use  the  product  to  maximize  their  time  spent  in  class. 

When  an  intervention  is  used  in  different  ways,  there  is  a  risk  that  the  intervention 
outcomes  will  not  meet  the  original  intent  or  plan.  During  the  Analysis  and  Design  phases, 
interventions  are  often  conceptualized  based  on  the  intent  to  address  an  identified  training  gap. 

If  the  intervention  is  used  in  a  different  manner,  there  is  a  risk  that  this  intent  will  not  be  met. 

One  best  practice  involved  training  developers  who  observed  the  instructors  using  the 
product  in  the  classroom  and  provided  feedback  on  how  it  was  used.  This  monitoring  and 
feedback  practice  helped  to  minimize  the  risk  that  the  final  products  would  not  meet  the  intent. 
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Evaluation  and  Sustainment 


Evaluation.  Objective  student  performance  data  was  not  collected  for  these  products. 
Some  reaction  data  was  collected  through  course  critiques.  Without  hard  data  reflecting  training 
effectiveness,  the  organization  cannot  determine  the  ROI  of  the  intervention.  When  evaluations 
are  not  conducted  it  also  is  difficult  to  obtain  feedback  on  what  modifications  or  enhancements 
are  needed. 

Our  findings  show,  however,  that  subjective  data  were  gathered  through  informal  means. 
In  many  cases,  instructors  could  simply  “tell  the  student  used  the  product”  based  on  how  the 
student  performed  in  the  subsequent  lesson  or  how  they  performed  on  live  equipment.  Although 
from  subjective  data,  these  results  suggested  that  the  products  supported  the  learning  objectives. 

Product  updates.  Challenges  with  product  updates  depended  on  whether  or  not  the 
product  was  developed  in-house  or  by  a  contractor.  When  products  were  developed  by 
contractors  using  proprietary  software  or  software  that  in-house  developers  were  not  trained  on 
product  updates  were  delayed  or  could  not  be  performed  without  sustainment  contracts  in  place. 

If  students  use  products  that  are  outdated  (including  simulators),  there  is  a  risk  for 
negative  training,  a  potential  loss  in  student  engagement,  and  an  increase  in  student  frustration 
with  outdated  instructional  material. 

When  contractors  used  software  that  could  be  edited  in-house  (i.e.  html-based  Unity),  this 
allowed  for  an  in-house  maintenance  strategy  rather  than  a  reliance  on  contractors  to  make  any 
necessary  updates.  Ultimately,  an  investment  in  in-house  development  could  minimize  the 
abovementioned  risks  and  impacts. 


Themes  by  IMI  Level 


Level  2  IMI  Product  Themes 

Level  2  IMI.  These  products  require  the  user  to  recall  more  information  than  a  level  1 
IMI  product  (i.e.  a  lesson  in  linear  format).  Users  have  more  control  over  the  lesson  content 
through  icons  and  other  peripherals  and  are  generally  assessed  using  simple  assessment  items 
such  as  multiple  choice  questions.  Data  were  collected  on  seven  level  2  IMI  products. 

Analysis,  design,  and  development 

SME-related.  For  100%  of  the  products,  it  was  reported  that  it  was  critical  to  have  the 
right  SMEs  involved  at  the  right  phases  of  design,  development,  and  testing  products.  The  right 
SME  is  not  only  a  content  expert;  he  or  she  can  also  be  an  instructor,  instructional  designer, 
multimedia  expert,  actor,  or  videographer. 

For  5/7  products  (71%),  it  was  reported  that  SME  turnover  created  challenges  to  the 
timeline,  scope,  and  budget  of  the  training  product  development.  It  is  important  to  have  the  same 
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set  of  SMEs  available  throughout  the  product  development  cycle.  In  some  instances,  strategies 
were  mentioned  to  mitigate  this  challenge  such  as  having  multiple  SMEs  assigned  or  having 
thorough  documentation  throughout  the  design  and  development  process. 

Front-end  analysis.  For  6/7  products  (86%),  the  need  for  the  training  was  linked  to  a 
performance  concern  or  training  gap  identified  either  through  observation  or  testing  (e.g., 
students  were  reported  as  having  trouble  performing  actions  or  being  weak  in  certain  areas; 
portions  of  the  instruction  required  modification;  students  needed  more  opportunities  to  learn 
about  the  equipment  since  there  were  not  enough  simulators). 

Leadership  support.  For  4/7  products  (57%),  leadership  buy-in  from  the  onset  through 
implementation  was  vital  to  the  success  of  the  product.  Buy-in  includes  a  general  consensus  on 
the  need  for  the  product  and  the  vision  of  what  its  final  outcome  should  be.  It  was  also  reported 
that  leadership  changes  during  the  product  development  lifecycle  produced  challenges.  For 
example,  a  new  leader  could  come  in  with  a  new  idea  of  the  need  and  a  new  vision  for  the 
product,  and  the  training  developers  would  have  to  “start  over.” 

Implementation 

POI  integration.  For  6/7  products  (87%),  it  was  reported  that  they  were  not 
systematically  incorporated  into  the  course  POIs  after  development.  It  was  also  reported  that  the 
products  were  not  monitored  or  tracked  after  development  to  ensure  they  were  actually  being 
used  in  the  classroom.  In  fact,  there  was  an  uncertainty  as  to  whether  instructors  were  using  the 
products  or  were  even  aware  that  they  existed.  Despite  lack  of  POI  integration,  some  of  the 
products  were  used  by  instructors  who  saw  the  benefit  of  their  use  and  passed  this  information  on 
to  other  instructors. 

Instructor  training.  For  4/7  products  (57%),  instructor  training  on  how  to  use  the 
products  was  very  minimal.  An  example  of  instructor  training  included  a  brief  explanation  or 
quick  run  through  of  the  product  with  instructors.  Fevel  2  IMI  products,  however,  are  generally 
self-explanatory  as  they  often  involve  a  student  opening  an  eFeaming  application  from  their 
laptop  and  continuing  self-paced  until  completion. 

Product  use.  100%  of  the  products  were  reported  as  being  used  in  a  variety  of  ways  to 
support  the  learning  objectives.  Related  to  the  POI  integration  statistic  above  (87%  not 
systematically  incorporated  into  the  course  POI),  instructors  were  almost  obliged  to  use  the 
product  in  different  ways  since  the  POI  did  not  provide  guidance.  It  would  depend  on  the 
instructor’s  experience  with  the  product  and  the  need  of  the  students  as  to  whether  it  was  used  as 
self-directed  training,  study  time,  preview  or  review  of  course  material  or  used  in-class  or  as 
homework.  Specially,  6/7  products  (86%)  were  leveraged  as  a  preview  or  review  of  course 
material  to  support  familiarization  or  refresher  training. 

Evaluation  and  sustainment 

Evaluation/learning  outcomes.  For  6/7  products  (86%),  it  was  reported  that  no  formal 
evaluation  was  conducted  to  determine  the  effectiveness  or  effects  of  using  the  product  on 
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student  learning.  Without  data  on  those  formal  measures,  it  is  challenging  to  validate  the  effects 
of  the  products  on  student  performance.  However,  for  5/7  products  (71%),  subjective  data  were 
gathered  informally  from  students  and  instructors  (e.g.,  products  reinforced  student 
understanding  and  retention,  promoted  self-paced  learning,  appealed  to  students  with  different 
learning  preferences,  and  increased  engagement  and  student  ownership  of  learning). 

Product  update/continuous  improvement.  For  4/7  products  (57%),  the  need  for 
revisions  fell  into  two  categories:  (1)  errors  resulting  from  technological  glitches  or  resulting 
from  content  inaccuracy,  and  (2)  new  materials  development  or  material  updates.  For  5/7 
products  (71%),  there  were  practical  challenges  for  updating  the  products  (e.g.,  required 
collecting  the  products  to  load  the  new  version  and  redistribute),  technical  issues  (e.g.,  operating 
system  upgrades),  and  other  challenges  (e.g.,  asset  continuity,  manpower,  and  contract 
management).  It  was  reported  for  4/7  products  (57%)  that  there  were  means  to  collect  and 
monitor  user  feedback  to  ensure  products  remained  accurate  and  up-to-date.  This  would  allow 
for  timely  and  continual  improvements. 

Across  the  product  development  cycle.  A  common  theme  emerged  across  all  ADD  IE 
phases  regarding  manpower  and  resources  for  Level  2  IMI  products.  Resource  constraints  which 
included  cost,  time,  expertise,  and  manpower  had  an  impact  on  the  type  of  product  developed  for 
the  course.  This  theme  was  reported  for  5/7  products  (71%). 

Level  3  IMI  Product  Themes 

Level  3  IMI.  These  products  require  the  user  to  recall  more  complex  information  than 
level  1  or  2  IMI  products.  Users  have  even  more  control  over  the  lesson  content  or  scenario  than 
a  Level  2  IMI  through  peripherals  such  as  light  pen,  touch  screen,  track  ball,  or  mouse.  Date 
were  collected  for  nine  level  3  IMI  products. 

Analysis,  design,  and  development 

SME-related.  Lor  7/9  products  (78%),  it  was  reported  that  there  was  adequate  SME 
support  throughout  design  and  development.  However,  it  was  noted  in  four  responses  that  there 
were  challenges  in  gaining  access  to  the  right  SMEs  or  that  there  was  a  lack  of  consistency  in 
SMEs  or  SME  input.  For  example,  there  were  difficulties  in  keeping  the  same  SMEs  involved 
throughout  the  process. 

Front-end  analysis.  For  8/9  products  (89%),  it  was  reported  that  there  was  a  reason  for 
product  development;  however,  the  reasons  varied.  For  three  products,  the  reason  was  associated 
with  a  leadership  request.  For  two  products,  the  reason  was  to  address  a  training  gap  between  the 
“walk”  and  “run”  phases,  i.e.  providing  opportunity  to  practice  before  live  exercises.  For  two 
products,  the  reason  was  aligned  with  the  ALM  tenet  of  rendering  the  training  more  engaging  to 
students.  For  one  product,  the  reason  was  to  address  logistical  limitations  of  the  training  (allow 
all  students  to  view  the  same  content  at  once). 

For  6/9  products  (67%),  it  was  reported  that  a  front-end  analysis  was  conducted; 
however,  the  methods  for  conducting  the  analysis  varied.  For  some  products,  the  course  was 
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observed  by  the  training  developer.  For  other  products,  the  training  developers  gathered  input 
from  instructors  and  students.  In  some  cases,  a  critical  task  analysis  was  performed. 

For  6/9  products  (67%),  it  was  reported  that  collaboration  and  consensus  among 
stakeholders  regarding  the  need  for  and  requirements  of  the  training  product  was  important  and  if 
lacking  led  to  issues  including  vague  requirements,  a  failure  to  explain  the  need  for  the  product 
to  the  developers,  difficulty  with  technology  tool  selection,  allowing  the  customer  to  dictate  what 
should  be  developed,  lack  of  clarity  regarding  product  ownership,  and  lack  of  input  from 
technology  team  early  on  in  the  design  process. 

Technology  specifications.  For  5/9  products  (56%),  the  products  were  used  on  multiple 
devices  or  browsers.  For  two  of  the  products,  frustration  resulted  from  a  lack  of  consistency  in 
tool  navigation  and  graphics  display  across  multiple  devices  or  browsers.  Responses  for  two 
products  indicated  a  desire  to  use  a  device-agnostic  approach  to  the  development  of  the  training 
product,  thus  reducing  this  type  of  frustration. 

Implementation 

POI  integration.  For  5/9  products  (56%),  it  was  reported  that  they  were  not  formally 
incorporated  into  the  course  POIs  after  development.  Despite  a  lack  of  POI  integration,  it  was 
reported  that  these  products  were  being  used  by  instructors  on  their  own  initiative  to  support 
course  learning  objectives. 

Instructor  training.  For  6/9  products  (67%),  no  formal  instructor  training  was  provided. 
In  some  cases,  instructors  were  given  opportunities  to  familiarize  themselves  with  the  product 
before  classroom  use. 

Product  use.  For  5/9  products  (56%),  it  was  reported  that  the  products  were  used  in  a 
variety  of  ways  to  support  the  learning  objectives.  Product  use  depended  on  the  instructors’ 
experiences  with  the  product  and  the  need  of  the  students  (e.g.,  whether  it  was  used  as  preview 
(prerequisite)  training,  classroom  instruction,  homework  assignment,  refresher  training,  or  as  a 
check  on  learning/assessment).  Three  of  the  products  were  used  for  both  individual  and  group 
activities. 

Instructor  use.  For  5/9  products  (56%),  the  responses  reflected  the  effects  of  using  the 
products  on  instructors’  workloads.  For  two  products,  reasons  for  increases  in  instructors’ 
workloads  included  the  use  of  the  product  itself,  the  instructor’s  lack  of  familiarity  with  the 
product,  and  the  added  administrative  requirements  associated  with  the  product.  For  one 
product,  instructor  workload  increased  at  the  onset  but  gradually  decreased  over  time.  For  one 
product,  instructor  workload  decreased  because  students  asked  fewer  questions.  For  one 
product,  there  was  no  difference  in  instructor  workload  as  a  result  of  using  the  product. 
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Evaluation  and  sustainment 


Evaluation/learning  outcomes.  For  100%  of  the  products,  it  was  reported  that  no  formal 
evaluation  occurred  to  determine  the  effectiveness  or  effects  of  using  the  product  on  student 
learning.  However,  for  6/9  products  (67%),  the  results  from  subjective  data  indicated  that  there 
was  a  positive  influence  on  student  learning  and  performance.  For  example,  for  5/9  products 
(56%),  it  was  reported  that  the  product  provided  a  good  visual  of  the  equipment,  allowing 
students  to  retain  information  and  perform  well  on  the  live  equipment. 

For  8/9  products  (89%),  students  received  feedback  on  their  performance  in  the  tool  via 
instructor  critiques  or  assessment  results  produced  by  the  tool  in  an  After  Action  Review  (AAR) 
format. 


Product  update/continuous  improvement.  For  6/9  products  (67%),  the  results  reflected 
constraints  for  product  updates.  Products  that  were  developed  in-house  faced  time  constraints 
while  products  developed  by  contractors  faced  issues  with  contracts  and  funding,  access  to 
support  personnel,  and  access  to  source  codes. 

For  5/9  products  (56%),  it  was  reported  that  the  selection  of  software  used  to  build  the 
technology  product  was  the  main  factor  for  updates  or  revisions  post-development. 

Across  the  product  development  cycle.  A  common  theme  emerged  across  all  ADD  IE 
phases  regarding  manpower  and  resources  for  Level  3  IMI  products.  For  5/9  products  (56%),  it 
was  reported  that  there  was  a  need  (or  preference)  for  in-house  development  and  sustainment  yet 
lack  of  manpower  and  resources  to  support  that. 

Level  4  IMI  Product  Themes 

Level  4  IMI.  These  products  involve  a  more  in-depth  recall  of  larger  amounts  of 
information  as  compared  with  the  other  IMI  levels.  The  user  has  an  increased  level  of  control 
over  the  lesson  material  and  scenarios.  Every  possible  subtask  is  analyzed  and  presented  with 
full,  on-screen  interaction,  similar  to  the  approach  used  in  aircraft  simulator  technology.  Data 
were  collected  for  five  level  4  IMI  products. 
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Analysis,  design,  and  development 


SME-related.  For  3/5  products  (60%),  it  was  reported  that  accessing  the  “right’  SMEs 
and  having  the  ability  to  leverage  the  same,  small  set  of  SMEs  was  vital  to  product  development 
cycle.  A  “right”  SME  reflects  individuals  with  relevant  content  expertise  and  familiarity  with 
all  details  (especially  with  terrain,  formation,  movement,  etc.)  Two  of  these  three  products 
reported  challenges  with  SMEs,  notably  that  high-performing  SMEs  were  sought  after  and 
obtained  by  other  organizations. 

Implementation 

POI  integration.  Eighty  percent  of  the  products  (4/5)  were  used  by  instructors  as 
optional  tools  during  a  course  to  reinforce  student  learning  and  engagement.  For  three  of  the 
products,  there  was  a  lack  of  adequate  training/communication  with  instructors  on  how  to 
integrate  the  tool  into  the  course  POIs. 

Instructor  training.  For  3/5  products  (60%),  it  was  reported  that  instructor  training  was 
required;  however,  the  amount  of  training  varied.  For  one  product,  a  significant  amount  of 
training  was  needed  in  order  to  show  instructors  how  to  bring  up  scenarios  and  inject  faults.  For 
two  of  the  products,  there  was  a  small  learning  curve  as  the  products  were  more  intuitive  in 
nature.  It  is  important  to  point  out  that  simulators  resemble  the  equipment;  therefore,  it  may  be 
inferred  that  instructors  needed  minimal  training.  Yet,  some  of  the  products  required  data  to  be 
pulled  from  simulators  or  required  instructors  to  use  monitoring  stations  for  performance 
observation.  Instructors  were  trained  informally  on  these  functions  by  other  instructors. 

Product  use.  Eighty  percent  of  the  products  (4/5)  were  reported  as  versatile,  allowing 
instructors  to  customize  and  adapt  scenarios  to  meet  different  learning  objectives  and  informally 
assess  student  performance.  Many  of  these  products  were  also  available  for  student  use  at  their 
home  station. 

Eighty  percent  of  the  products  (4/5)  were  used  by  instructors  as  the  “walk”  phase  of  the 
Army’s  crawl,  walk,  run  training  method  so  students  were  able  to  practice  before  using  the  actual 
equipment  and  during  class  “down  time”.  For  example,  the  products  were  used  to  get  struggling 
students  up  to  speed  with  additional  practice  or  to  provide  students  with  additional  practice  while 
they  were  waiting  to  train  on  the  live  equipment.  Products  were  also  used  as  reach-back 
(sustainment)  training. 

Instructor  use.  Eighty  percent  of  the  products  (4/5)  provided  minimal  feedback  to 
students  on  their  performance;  therefore,  instructors  had  to  monitor  students  using  the  product 
and  provide  feedback  and  guidance. 

For  3/5  products  (60%),  there  was  little  to  no  impact  on  instructor  workload.  In  some 
cases,  the  impact  on  the  instructor  workload  was  a  zero-sum  gain;  it  was  reported  as  the  same 
amount  of  work,  just  different. 
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Evaluation  and  Sustainment 


Evaluation/learning  outcomes.  For  4/5  products  (80%),  it  was  reported  that  no  objective 
or  hard  data  existed.  However,  instructors  and  students  reported  that  the  use  of  the  tools  had  a 
positive  impact  on  student  performance.  Instructors  indicated  that  they  could  see  the  difference 
with  live  equipment  performance  based  on  use  of  the  products. 

For  4/5  products  (80%),  it  was  reported  that  the  accuracy  of  simulation,  as  it  resembles 
live  equipment,  is  very  important  to  achieving  targeted  learning  outcomes. 

It  was  reported  for  3/5  products  (80%)  that  the  tools  impacted  ROI  by  reducing  training 
time  on  the  live  equipment.  Some  of  the  examples  of  this  ROI  included  a  reduction  of  wear  and 
tear  on  the  live  equipment  as  well  as  reduction  in  fuel  costs  as  it  pertained  to  time  needed  in 
aircrafts  (since  students  were  already  prepared  and  did  not  need  to  train  on  fundamental  tasks). 

Product  update/continuous  improvement.  For  4/5  products  (80%),  the  need  to  maintain 
the  product  fell  mostly  into  two  categories:  software  upgrades  and  reliability  issues  such  as  bugs 
causing  system  to  malfunction  or  shut  down.  The  results  for  three  of  the  products  indicated  the 
importance  of  documenting  bugs  or  glitches,  as  this  assists  with  the  maintenance  process. 

Across  the  product  development  cycle.  A  common  theme  emerged  across  all  ADD  IE 
phases  regarding  manpower  and  resources  for  Level  4  IMI  products.  For  3/5  products  (60%),  it 
was  reported  that  the  lack  of  funding  or  an  ongoing  maintenance  contract  was  found  to  be  a 
barrier  both  in  development  and  sustainment. 
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